Non-natural pattern identification for cognitive assessment

ABSTRACT

Methods, systems, and apparatus, including medium-encoded computer program products, for detection of cheating on a cognitive test. In one aspect, a method includes receiving first information concerning a person, the first information specifying the person&#39;s responses, and lack thereof, for items of a cognitive test administered to the person, wherein the cognitive test includes multiple item-recall trials used to assess cognitive impairment; analyzing the first information using a classification algorithm trained on second information concerning a group of people to whom the cognitive test has been administered, the classification algorithm generated in accordance with a first part and a second part, the first part distinguishing between cheaters and non-cheaters, and the second part distinguishing between impaired cheaters and non-impaired cheaters; and identifying, based on the analyzing, the person as a cheater requiring a verification test to confirm an initial result of the cognitive test.

CROSS REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of the priority of U.S. ProvisionalApplication Ser. No. 61/036,881, filed Mar. 14, 2008 and entitled“Non-Natural Pattern Identification During Cognitive Assessment”.

BACKGROUND

This specification relates to assessing the cognitive function of aperson to whom a cognitive test has been administered, and in particularto detection of cheating on a cognitive test.

Various techniques have been used to measure the cognitive function of aperson. For example, the National Institute of Aging's Consortium toEstablish a Registry of Alzheimer's Disease (CERAD) has developed a tenword list as part of the Consortium's neuropsychological battery. TheCERAD word list (CWL) test consists of three immediate-recall trials ofa ten word list, followed by an interference task lasting severalminutes, and then a delayed-recall trial, with or without adelayed-cued-recall trial. The CWL is usually scored by recording thenumber of words recalled in each of the four trials. A single cutoffscore for the delayed-recall trial, with or without adjustment fordemographic variables, is typically used to determine whether cognitiveimpairment exists.

Some have proposed various improvements to the CWL. In addition, the CWLand the improvements thereof have been used to provide memoryperformance testing services, via the Internet, to clinicians in dailypractice. Such services allow rapid testing of individual patients andreporting on the results of such testing. Previous reports forindividual cognitive performance test results have included a statementof whether the patient has been found to be normal or to have cognitiveimpairment.

Furthermore, in the long-term care insurance industry, individualsapplying for a policy must typically be determined to not have cognitiveimpairment or dementia due to Alzheimer's disease or a related disorder(ADRD). Insuring such an impaired individual can result in a typicalclaims cost of more than $200,000 per case, assuming four years ofclaims payments. The insurer therefore wishes to avoid insuringapplicants who already have ADRD. For this reason, insurers payunderwriters to administer cognitive testing that is both sensitive todetect mild cognitive impairment, as well as specific to correctlyidentify normal aging. To reduce the costs of such cognitive testing,insurers are increasingly testing applicants over the telephone.

SUMMARY

This specification describes technologies relating to assessing thecognitive function of a person to whom a cognitive test has beenadministered, and in particular to detection of cheating on a cognitivetest.

In general, an aspect of the subject matter described in thisspecification can be embodied in one or more methods that includereceiving first information concerning a person, the first informationspecifying the person's responses, and lack thereof, for items of acognitive test administered to the person, wherein the cognitive testincludes multiple item-recall trials used to assess cognitiveimpairment; analyzing the first information using a classificationalgorithm trained on second information concerning a group of people towhom the cognitive test has been administered, the classificationalgorithm generated in accordance with a first part and a second part,the first part distinguishing between cheaters and non-cheaters, and thesecond part distinguishing between impaired cheaters and non-impairedcheaters; and identifying, based on the analyzing, the person as acheater requiring a verification test to confirm an initial result ofthe cognitive test. Other embodiments of this aspect includecorresponding systems, apparatus, and computer-readable media encodingcomputer program product(s) operable to cause data processing apparatusto perform the operations.

These and other embodiments can optionally include one or more of thefollowing features. The classification algorithm can be configured tocheck for cheating strategies characteristic of persons with Alzheimer'sdisease or a related disorder (ADRD). The classification algorithm canbe selected to maximize sensitivity while minimizing reduction inspecificity, which preserves a high negative predictive value whilemaintaining a low misclassification rate for impaired cheaters.Moreover, the group of people can include a first sample and a secondsample, where the method further includes: analyzing data in the firstpart for the first sample to identify a subset of variables thatdiscriminate between cheaters and non-cheaters, and validating resultsin the first part using the second sample; and analyzing, in the secondpart, data of persons identified as cheaters in the first part toidentify a subset of variables that discriminate between impairedcheaters and non-impaired cheaters.

The subset of variables that discriminate between cheaters andnon-cheaters can include education and multiple numbers corresponding toitems recalled on two or more of the multiple item-recall trialsincluding a delayed free recall trial, and the subset of variables thatdiscriminate between impaired cheaters and non-impaired cheaters caninclude age, gender, education and a number corresponding to itemsrecalled on at least one of the multiple item-recall trials. Themultiple item-recall trials can include word recall tests of memory, andthe analyzing can include distinguishing between impaired andnon-impaired individuals based on a total number of words recalledacross the trials. The analyzing can include evaluating a probability ofan order of items recalled by the person given probabilities of recallpatterns for the group of people.

A system can include a user device; and one or more computers operableto interact with the device and to perform operations including those ofthe method discussed above. The one or more computers can include aserver system operable to interact with the device through a datacommunication network, and the device can be operable to interact withthe server as a client. Moreover, the device can include a userinterface device, the one or more computers can include the userinterface device, and the operations can further include outputting anindication of the identified person to a device including acomputer-readable medium.

Particular embodiments of the subject matter described in thisspecification can be implemented to realize one or more of the followingadvantages. Cheaters can be readily detected from information specifyinganswers on a cognitive test that includes multiple item-recall trialsused to assess cognitive impairment. Analysis of the word order acrossmultiple item-recall trials can significantly increase the ability todetect cheaters. Moreover, substantial savings can be realized bydetecting 1) test cheaters so as to avoid inappropriate insurancepolicies being issued; 2) cheating by insured individuals attempting tofail a test to obtain long-term care insurance claims benefits early; 3)cheating by individuals attempting to do worse on a test to becomeeligible for a clinical trial; and 4) cheating by professionalsattempting to pass a cognitive test that is required for their continuedemployment.

Moreover, the systems and techniques described can be implemented foruse in other scenarios in which individuals are tested on theircognitive abilities. Examples include taking entrance exam tests foradmission to professional schools, taking tests to qualify for abenefit, an insurance policy or a clinical drug trial, and competitionsfor prizes, prestige or recognition by others, or other scenarios whereindividuals have an incentive to do well on a given test.

The details of one or more embodiments of the subject matter describedin this specification are set forth in the accompanying drawings and thedescription below. Other features, aspects, and advantages of theinvention will become apparent from the description, the drawings, andthe claims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows an exemplary system used to identify cheaters on acognitive test.

FIG. 2 shows an exemplary process used to identify cheaters on acognitive test.

FIG. 3A shows an exemplary process of generating a classificationalgorithm to identify cheaters on a cognitive test.

FIG. 3B shows an exemplary process of detecting cheating on telephoneadministered cognitive testing of a long-term care insurance applicant.

FIG. 4 shows another exemplary system used to identify cheaters on acognitive test.

DETAILED DESCRIPTION

FIG. 1 shows an exemplary system 100 used to identify cheaters on acognitive test. A data processing apparatus 110 can includehardware/firmware and one or more software programs, including acognitive test responses analysis program 120. The cognitive testresponses analysis program 120 operates in conjunction with the dataprocessing apparatus 110 to effect various operations described in thisspecification. The program 120, in combination with the varioushardware, firmware, and software components of the data processingapparatus, represents one or more structural components in the system,in which the algorithms described herein can be embodied.

The program 120 can be an application for performing analysis on datacollected to assess the cognitive function of a subject. An applicationrefers to a computer program that the user perceives as a distinctcomputer tool used for a defined purpose. An application can be builtentirely into an operating system or other operating environment, or itcan have different components in different locations (e.g., a remoteserver). The program 120 can include or interface with other softwaresuch as database software, testing administration software, dataanalysis/computational software, and user interface software, to name afew examples. For example, the program 120 can include or interface withsoftware that collects data and analyzes the cognitive function orotherwise assesses the brain condition of a person.

Interface software can also be included that operates over a network tointerface with other processor(s), such as in a computer used by a testtaker or test administrator. For example, the program 120 can includesoftware methods for inputting and retrieving data associated with acognitive assessment test, such as score results, or demographic data.Such a cognitive assessment test can be administered using the program120, or the program 120 can be used to collect and analyze data from acognitive assessment test administered in another manner, such as anin-person or a test administered by telephone. In addition, the program120 can also effect various analytic processes, which are describedfurther below.

The data processing apparatus includes one or more processors 130 and atleast one computer-readable medium 140 (e.g., random access memory,storage device, etc.). The data processing apparatus 110 can alsoinclude one or more user interface devices 150. User interface devicescan include display screen(s), keyboard(s), a mouse, stylus, modems orother networking hardware/firmware, or any combination thereof to name afew examples. The subject matter described in this specification canalso be used in conjunction with other input/output devices, such as aprinter or scanner. The user interface device can be used to connect toa network 160, and can furthermore connect to a processor or processors170 via the network 160 (e.g., the Internet).

Therefore, a user of the analysis program 120 does not need to be local,and may be connecting using a web browser on a personal computer, orusing other suitable hardware and software at a remote location. Forexample, a clinician at a testing center can access a web interface viathe remote processor 170 in order to input test data for a cognitivetest. The test data can be the results of an already administered test,or the test data can be the information exchanged when actuallyadministering the cognitive test using a network based testing system.In any event, data can be transmitted over the network 160 to/from thedata processing apparatus 110. Furthermore the clinician can input testdata and retrieve analysis based on that data or other data stored in adatabase. Note that the data processing apparatus 110 can itself beconsidered a user interface device (e.g., when the program 120 isdelivered by processor(s) 170 as a web service).

The system 100 can be used to analyze data from various types ofcognitive function or brain condition assessment tests. The followingdescription provides extensive details with respect to detecting testcheaters in the context of persons with Alzheimer's disease or a relateddisorder (ADRD). However, the systems and techniques described can beimplemented for use in other scenarios in which individuals are testedon their cognitive abilities. Thus, the systems and techniques describedcan be used in many different contexts to detect cheating among personstaking neuropsychologic tests.

In the context of ADRD, if an individual already has ADRD, they have astrong incentive to try and obtain a long-term care insurance policy tocover the costs of their care. If they take the test over the telephone,it is easy to cheat by taking notes on what items they will be asked torecall later. Such cheating could result in an ADRD individual beingissued a long-term care insurance policy that will cost the insurerupwards of $200,000.

Because of the increased use of telephone testing when applying for along-term care insurance policy, more individuals already affected byADRD will be able to cheat and pass the test to receive a long-term careinsurance policy. Based on results discussed below, it is estimated thatof 100,000 applicants for long-term care insurance annually tested bytelephone, approximately 1,140 individuals with ADRD will cheat and passtesting to receive a policy. This can result in annual claims costs ofaround $228 million or more to the long-term care insurance industry. Itis therefore advantageous for insurers to devise strategies to detectcheating when applicants are tested by telephone.

The strategies that people use to cheat on a cognitive test are limited.In the case of cheating by the test taker, cheating fundamentallyinvolves the use of a strategy of providing answers without an examinerrealizing that the person being tested is obtaining them in violation ofthe rules of the test. For example, in testing memory over thetelephone, a person may write down all the words they are asked toremember and recall later. When asked to recall the words, they may thenread the list back to the examiner. The way that the person reads thewords back to the examiner constitutes their cheating strategy. Oneperson's cheating strategy may be to read back the words in the orderthey were presented. Another's cheating strategy may be to read back thewords in their reverse order. Another's cheating strategy may be to readback the list of words in a random order.

In deciding whether a person is cheating, one can compare their memoryperformance to well characterized patterns observed among non-cheatingindividuals. These normal patterns of memory retrieval are non-intuitiveand quite distinct from most cheating strategies conjured up byindividuals. These normal patterns of memory performance consist of (1)changes in total numbers of words recalled across successive learningand testing trials, and (2) stereotypical orderings of the wordsrecalled in each trial.

FIG. 2 shows an exemplary process 200 used to identify cheaters on acognitive test. First information is received (210) concerning a person,where the first information specifies the person's responses, and lackthereof, for items of a cognitive test administered to the person. Theinformation can be from a previously administered test or from a testthat is currently being administered. Nonetheless, the exemplary processdescribed in connection with FIG. 2, and other implementations of themore general concepts underlying this exemplary process, are notpracticed on the human body since such processes do not themselvesinvolve an interaction necessitating the presence of the person.

The test can be administered in-person, over the phone, through acomputer network (e.g., through the Web over the Internet), or inanother manner Note that for in-person administration of the test, thepresent system and techniques can still provide benefits in thatcheating can be detected in the event that the test administrator is incollusion with the test taker or the test administrator is notsufficiently trained to accurately detect cheating that occurs during anin-person administration of the cognitive test.

The cognitive test can include multiple item-recall trials, which canfurther include at least one item common to a subset of the recalltrials, the subset including at least two of the recall trials.Alternatively, the multiple item-recall trials can have no items incommon. In general, the full set of information in the test should berecorded, including all components of the test and all subjectresponses. The information can be received (210) from a database, anetwork or web-enabled device, a computer readable medium, or a standardinput output device on a computer system, to name just a few examples.The cognitive test can include a test of attention and recall, and thetest components can include items (e.g., words) to be recalled in one ormore trials. For example, a test of attention and recall can include theCERAD word list (CWL) and/or other lists of words or items.

The CWL is a test of immediate and delayed free recall and delayed cuedrecall that was developed by the National Institute of Aging CERADcenters in the 1980s. There are three learning trials in which thesubject is presented each word in the list and repeats it, then at theend of the list, recalls as many words as they can. The subject is notinstructed to recall the words in the order they are presented, butrather to recall as many words as they can immediately after beingpresented the list of ten words. They are also instructed that a fewminutes after the third learning trial they will again be asked torecall as many of the words as they can without another presentation ofthe words. The words are presented in a different order for eachlearning trial. The number of words correctly recalled is recorded foreach of the three learning trials. After the third learning trial, aninterference task that distracts the subject from rehearsing the wordlist (e.g., a test of executive function, which can consist of thesubject being asked to select which of three animals is most differentfrom the other two, and is given 12 such triads of animals to makedecisions upon) is given over a period of two to five minutes. After theinterference task, the subject is asked to recall as many of the tenwords as they can (delayed free recall trial). The number of wordscorrectly recalled is recorded. After the delayed free recall trial, thesubject is given a delayed recognition task. The subject is presentedthe ten CWL words intermixed with ten distracter words. For each word,the subject is asked whether it was one of the CWL words, and thesubject's response (yes or no) is recorded.

Since the words of the trials are already known, the first informationneed not specify the words themselves, but rather just whether or not agiven word was recalled. For example, eight word lists can be used, witheach word list including ten words for learning and recall, plus tenmore words for delayed-cued-recall. Four trials can be employed in thecognitive test, where one of the eight word lists can be selected foruse in the test. The first set of ten words from the list can be used inthe immediate and delayed free recall trials (and the words of the listcan be presented in the same order in each trial or in a differentorder), and the second set of ten words can be used as the distracterword list for the delayed-cued-recall trial. The first information caninclude an eighty column binary score (i.e., an eighty bit vector) thatcorresponds to the responses received on the immediate and delayed freerecall trials of the cognitive test. Each bit in this example indicateswhether a corresponding word from a trial was recalled, or whether thecorresponding word from the trial was not recalled.

For example, an eighty columns wide binary indicator matrix can bedefined as follows. Each word in each trial can occupy 2 columns. Thefirst column can be assigned a 1 if the word in the trial was recalledand a 0 if it was not recalled. The second column can be assigned a 0 ifthe word in the trial was recalled and a 1 if it was not recalled. Eachtrial with ten words thus occupies twenty columns for a total of eightycolumns for the four free recall trials of the word list trials. Withthis arrangement, the binary indicator matrix gives a row total offorty, which permits the determination of an optimal column score for aword when it was recalled in a trial, as well as a different optimalcolumn score when that word was not recalled in a trial.

The words in each word list can be linguistically and statisticallyequivalent. The words on each distinct list can have the same level ofintra-list associability and usage frequency. Each list of words canhave the same level of associability and usage frequency with each andevery other list of words. For example, the eight word lists used can beas shown in Table 1:

TABLE 1 Word Lists List 1 List 2 List 3 List 4 List 5 List 6 List 7 List8 W1 BUTTER BEDROOM CAKE CLOCK BIBLE OAK JAZZ BAT W2 ARM DOWN PARK SCALEFEMALE RANK BUS SAFETY W3 SHORE MESSAGE WISDOM THREAT LEGEND TASTE LIDCOPY W4 LETTER BIRTHDAY MARRIAGE SPORT STAMP SPRING CRITIC ROOF W5 QUEENWIND REST SPACE TOOTH BRAND DARK ACTOR W6 CABIN TRUCK NOTICE LAYER FATPROJECT OWNER VISIT W7 POLE LEADER BOAT AMOUNT GLOVE SERVANT GUEST POOLW8 TICKET HAT PLANET FLOOD LECTURE CUP WEATHER GRIEF W9 GRASS BARN KNEEDOUBLE BEAST LIST PEACE SLEEVE W10 ENGINE SOCK TELEPHONE RESPECT AGENTPLAIN BASE OUTCOME D1 CHURCH WINTER BLANKET TOUCH SHOW CAMP MUSCLE DANCED2 COFFEE BAG VEIN FLOOR CASH BATHROOM ORGAN REGION D3 DOLLAR BLUE SHAPELEATHER HELICOPTER OIL WEDDING SMOKE D4 FIVE ROOT NEWSPAPER ARROW FLOWEREARTH WOOD BLADE D5 HOTEL TRAIL MISSION KID NUT BEEF SUPPORT STRESS D6MOUNTAIN SEED WATCH BUCKET SILVER LUNCH PARKING LIMIT D7 SLIPPER HEARTLIGHT CONFLICT BOTTLE PORTRAIT BRANCH TRIAL D8 VILLAGE SOUP PINT DUSTLOYALTY HOST PHOTO PENCIL D9 STRING NOISE CYCLE PRESSURE LOAD STRUGGLEVERSE WIFE D10 TROOP CREATURE MOUTH SPELL DECADE RIDE LOUNGE PLAYER W#:10 Word List used in learning trial to be recalled D#: Used inDelayed-Cued-Recall Trial along with the 10 Word List

The word lists can be used in different parts of a test (e.g., thedistracter and learning word lists can be interchanged). Moreover, thewords in each word list can be presented in the same order or differentorder. For example, a shuffled order can be employed over multipletrials, such as in the CERAD or the ADAS-Cog (Alzheimer's DiseaseAssessment Scale-cognitive subscale) cognitive assessment tools.ADAS-cog consists of eleven tasks measuring different cognitivefunctions. The ADAS-Cog word recall test has the same general method oftest administration as the CWL. Note that the ADAS-Cog but does not usethe 10-word list for cued recall that is used in the immediate anddelayed free recall trials. It has its own set of words for that.

In general, the words in each word list should have the same difficultyof being recalled as the other words on that list, as well as the wordsin the other lists. For each learning trial, the words can be presentedin the same order or in different order. It will be appreciated thatother data formatting approaches, as well as other cognitive tests andtest components, are also possible.

Other cognitive assessment tests can include, but are not limited toother multiple word recall trials, other recall or cued recall tests ofverbal or non-verbal stimuli, tests of executive function, includingtriadic comparisons of items, (e.g., deciding which one of three animalsis most different from the other two), tests of judgment, similarities,differences or abstract reasoning, tests that measure the ability toshift between sets or perform complex motor sequences, tests thatmeasure planning and organizational skill, tests of simple or complexmotor speed, tests of language abilities including naming, fluency orcomprehension, tests of visual-perceptual abilities including objectrecognition and constructional praxis. In one implementation, subjectsare asked to recall the nine animals that were used for the triadiccomparisons interference task that was given between the third learningtrial and the delayed free recall trial. This delayed free recall ofanimals differs from that of the wordlist in that the subject is notasked to remember the animal names to recall them later, and thereforemay not write them down if they are cheating. Differences between thedelayed free recall of the animals and of the wordlist can help identifytest cheaters. Examples of recorded data can include the words recalled,the words not recalled, the order of the words recalled, time delaybefore recall, the order in which intrusions and repetitions arerecalled, and various aspects of test performance. Moreover, thecognitive test can include one or more trials performed to determinespecific cognitive functions such as physical (e.g. orientation orhand-eye coordination) or perception based tests. Additional informationcan be obtained in order to classify the score, such as demographicinformation, or the date(s) of test administration, to name just twoexamples.

In any case, the first information is analyzed (220) using aclassification algorithm trained on second information concerning agroup of people to whom the cognitive test has been administered. Theclassification algorithm is generated in accordance with a first partand a second part, where the first part distinguishes between cheatersand non-cheaters, and the second part distinguishes between impairedcheaters and non-impaired cheaters. Thus, the classification algorithmcharacterizes the ability to detect cheating among normal and impairedindividuals (e.g., those with ADRD). In the context of multipleitem-recall trials, this can be done based on the total numbers of wordsrecalled across trials (e.g., the total scores of the various trials ofthe CERAD wordlist), upon ratios of total scores for different pairs oftrials, upon the recall of a given word or set of words across trials,upon the order in which a given word or set of words is recalled acrosstrials, or based upon the orderings of words recalled per trial todetect different types of cheaters.

The classification algorithm can be configured to check for cheatingstrategies characteristic of normal persons and persons with someimpairment, such as ADRD. One reason to think that individuals impairedwith ADRD may have a different cheating strategy than normal individualsis that executive function is affected early in ADRD. Individuals useexecutive function to select a cheating strategy. The cheating strategyfor impaired ADRD individuals may therefore be simpler or differ in someother way than that for normal individuals. Identification of a subsetof cheating strategies that characterize most impaired ADRD individuals(or individuals with other executive function impairments) can improvethe accuracy of cheating detection and reduce the costs required tore-test suspected cheaters in a setting where they cannot cheat.

The person can thus be identified (230), based on the analyzing (220),as a cheater requiring a verification test to confirm an initial resultof the cognitive test. The verification test can be the same cognitivetest administered using a different protocol. For example, when the testis initially administered by phone, the verification test can be anin-person or video-monitored administration of the same cognitive testwith a different set of similar stimuli, such as a different, butequivalent, wordlist. Alternatively, the verification test can be adifferent test than the initially administered cognitive test. Forexample, if the test is initially administered by phone, theverification test could consist of a second test administered by phonein which the stimuli, such as tones, cannot be written down.

FIG. 3A shows an exemplary process 300 of generating a classificationalgorithm to identify cheaters on a cognitive test. In a first part ofalgorithm generation, data can be analyzed to identify a subset ofvariables that discriminate between cheaters and non-cheaters, andresults in the first part can be validated (310). The group of peopleused to train the classification algorithm can include a first sampleand a second sample, the data analyzed in the first part can come fromthe first and second samples, and the results in the first part can bevalidated using a different mixture of the second sample, such asdescribed in further detail below.

In a second part of algorithm generation, data of persons identified ascheaters in the first part can be analyzed to identify a subset ofvariables that discriminate between impaired cheaters and non-impairedcheaters (320). The persons identified as cheaters can be from thesecond sample, the first sample, or both. The analysis operations (310,320) can be performed multiple times using different inputs to createmultiple variations of the classification algorithm. In any case, aclassification algorithm can be selected to maximize sensitivity whileminimizing reduction in specificity, which preserves a high negativepredictive value while maintaining a low misclassification rate forimpaired cheaters (330).

A detailed example is now discussed, in which total scores as well asratios of total scores of the various sub-tests of a memory test wereexamined. In this example, the first sample included 50 subjects with noevidence of cognitive impairment and normal job performance who wereadministered a memory test over the telephone twice. In this example,the subjects in the first sample were assigned at random to cheat oneither the first or the second test, and instructed to not cheat on theother test. Moreover, the subjects did not receive the same wordlist forthe two tests.

The second sample included 15,467 individuals applying for long-termcare insurance who took the memory test over the telephone and passedthe test. They were asked at the end of the test whether they hadwritten down any words. If they answered yes, they were classified asreported cheaters. They were also evaluated for suspected cheating basedon pre-established criteria, such as a highly suspicious pattern ofwords recalled. Applicants suspected of cheating were classified assuspected cheaters. Of these 15,467 individuals, 15,038 had completedata that allowed for full analysis. The sample was therefore restrictedto the 15,038 applicants with complete data.

A 30% sub-sample of the suspected and reported cheaters wereadministered the test a second time (N=847). This time, they wereadministered the test in-person to prevent cheating. Those applicantswho failed the in-person test were classified as impaired cheaters,while those who passed were classified as unimpaired cheaters.

Candidate Variables For Cheating Classification Algorithm—Candidatevariables that were examined to detect cheaters in parts 1 and 2 of thecheating algorithm included demographics (age, gender, education) andvariables assessing different aspects of the individual's cognitive testperformance. These cognitive variables included total numbers of wordsrecalled on each of the trials, the 6 ratios of the total numbers ofwords recalled for all possible pairs of four trials, the numbers ofwords recalled that were repetitions or intrusions (words not in thelist) in each trial, the delayed free recall of animals (a separatemeasure of memory performance in which there was no specific instructionto try and remember, later on, the 9 animals that are presented to thesubject—three at a time—as they are asked to select which animal is mostdifferent from the other two), and the ratio of the number of wordsrecalled on the delayed free recall from the 10-word list compared tothe delayed free recall of the 9 animals from the triadic comparisonstask.

Cheating Algorithm Part 1—Cheaters Versus Non-Cheaters: The trainingsample had 208 subjects and consisted of all sample 1 subjects plus anequal number of sample 2 cheater and non-cheater subjects. Stepwiselogistic regression was then performed on the candidate variables toremove ones that were clearly non-predictive. For the remainingvariables, a bootstrap procedure was then applied, in which two thirdsof the cheaters and two thirds of the non-cheaters in the trainingsample were randomly selected one thousand times, and logisticregression was performed each time. These thousand runs generatedbootstrapped confidence intervals of the coefficient values for each ofthe remaining candidate variables. Any candidate variable whosebootstrapped confidence interval included zero was removed from theclassification model. Variables included in the final part 1 modeldiscriminating cheaters from non-cheaters included education, the numberof words recalled on the first learning trial and on the delayed freerecall trial, the ratio of the numbers of words recalled on trial 2versus trial 1, the ratio of the corresponding numbers for the delayedfree recall trial versus learning trial 2, and the ratio of the numberof words recalled on delayed free recall of the wordlist versus that ofthe animals. Additional parameters that properly weighted theclassification algorithm of cheater versus non-cheater included theprior probability of cheating by applicants for long-term care policiesin the absence of other knowledge.

To validate the final part 1 algorithm, the set of the 847 sample 2subjects were classified using the algorithm, and a non-parametricreceiver operating characteristic (ROC) curve was used to estimateoverall accuracy. The ROC curve defines the overall accuracy as the areaunder the curve, and defines the possible pairs of sensitivity andspecificity values for discriminating cheaters from non-cheaters aspoints on the curve (regardless of whether the cheaters were normal orimpaired). Applicants from sample 2 who were classified as cheaters werethen submitted to part 2 of the algorithm to discriminate non-impairedcheaters from impaired cheaters.

The algorithm for part 1, which is applied to persons who had taken thecognitive test by telephone and were classified as normal, was asfollows: Y=−1n[β₀*(prevalence of not cheating)]+β₁*education+β₂*Trial 1Total+β₃*Delayed Free Recall Trial Total+β₄*(Trial 2 Total/Trial 1Total)+β₅*(Delayed Free Recall Trial Total/Trial 2 Total)+β₆*DelayedFree Recall Of Animals Total. Y is the score that is used, inconjunction with the true cheating status of each case, by a receiveroperating characteristic curve to discriminate cheaters fromnon-cheaters. The β_(i) coefficients are the regression weights used tomultiply by the scores of the various variables in the equation.

Cheating Algorithm Part 2—Impaired Cheaters Versus Normal Cheaters: A70% random sub-sample of the 847 applicants classified, by in-persontesting, as impaired cheaters or normal cheaters were analyzed usingstepwise logistic regression to look for variables discriminatingimpaired cheaters from normal cheaters. A similar set of candidatevariables from the memory test and demographic variables were submittedfor inclusion into the stepwise logistic regression. From this analysis,variables that were clearly non-predictors were excluded from the model.Bootstrap analysis was then performed using one thousand random samplesof equal numbers of impaired and non-impaired cheaters—forty three tofifty five per group—and logistic regression was performed with eachrandom sample to generate a bootstrapped confidence interval for eachvariable. Variables whose coefficient values had confidence intervalsthat included zero were excluded from the model. The remaining variablesin the model were run through a second iteration of this process so thatthey had to survive stepwise regression followed by bootstrappedconfidence interval estimation to remain in the final model for part 2.To validate the resulting final model for part 2, the full set of fiftyseven impaired cheaters were selected, and a 9% random sample of thenormal cheaters were selected and then classified using a non-parametricROC curve. This process was repeated twenty five times to obtain theoverall accuracy and confidence interval for part 2.

Variables that performed well in part 2 of the cheating classificationalgorithm were age, gender, education, the number of words recall on thethird learning trial, and the number of words recalled on the delayedfree recall of animals. Additional parameters that properly weighted theclassification algorithm of impaired cheater versus non-impaired cheaterincluded the prior probabilities of cheating by impaired andnon-impaired applicants for long-term care policies in the absence ofother knowledge.

The algorithm for discriminating impaired versus non-impaired cheaterswho had taken the cognitive test by telephone and were classified as apossible cheater was as follows: Y=−1 n[β₀*(prevalence of non-impairedcheater/prevalence of impairedcheater)]+β₁*age+β₂*gender+β₃*education+β₄*Trial 3 Total+β₅*Delayed FreeRecall of Animals Total. Y is the score that is used, in conjunctionwith the true status of each case, by a receiver operatingcharacteristic curve to discriminate impaired cheaters from non-impairedcheaters. The β_(i) coefficients are the regression weights used tomultiply by the scores of the various variables in the equation.

Such classification algorithm(s) can thus be used to detect cheating bya long-term care insurance applicant taking a cognitive test over thetelephone. FIG. 3B shows an exemplary process 350 of detecting cheatingon telephone administered cognitive testing of a long-term careinsurance applicant. Telephone based EMST (Enhanced Mental Skills Test)testing can be performed (355). Classification algorithm(s) can be run(360) using results of the testing. A check can then be made (365) todetermine whether to decline coverage, or proceed further.

If a normal determination is made (365), cheating versus no-cheatingalgorithm(s) can be run (370). A check can then be made (375) to assesswhether cheating has occurred. If not, the potential underwriting of aninsurance policy can be triggered. If cheating has likely occurred, theprocess can proceed further where impaired versus normal-cheateralgorithm(s) can be run (380). A check can then be made (385) to assesswhether a likely cheater is a likely impaired cheater. If not, thepotential underwriting of an insurance policy can be triggered. If thelikely cheater is likely an impaired cheater, a face to face (FTF) orother validation test can be given (390). A check can then be made (395)to assess impairment and either decline coverage or potentially triggerthe underwriting of an insurance policy.

A total of 18.9% of the 15,038 applicants were either reported cheaters(14.4%) or suspected cheaters (4.5%). Of the 30% sub-sample of 847reported and suspected cheaters re-tested in-person, 94% and 6% of themwere classified as normal cheaters and impaired cheaters, respectively.Compared to the 2.6% prevalence of impairment among persons notsuspected of cheating, cheaters are 2.3 times more likely to be impairedthan non-cheaters.

Table 2 below summarizes the savings in claims costs for the sample of15,038 applicants who were screened normally by telephone and wereevaluated for cheating using a subset of multiple versions of theclassification algorithm. The different versions correspond to differentcut points on the ROC curve used. The second and last columns of thetable are normalized to allow generalization of the findings to anysample size.

TABLE 2 Range of Costs and Cost Savings for Different Algorithm VersionsCurrent Claims Additional Cost Per ($201,500 per claim) Claims Saved ByNet Claims Net Claims Normal Tel Test Among All Telephone Detecting TelNormal Savings Due to Savings Per 1,000 Algorithm to Detect ImpairedNormal Applicants Impaired Cheaters Detecting Tel Normal TelephoneVersion Cheaters Without Cheating Algorithm ($201,500 per claim)Impaired Cheaters Normal Applicants V1_11.2_13.1  $(5.25) $(59,808,475)$20,292,161 $20,213,161 $1,344,139 V2_11.2_13.1  $(6.64) $(59,808,475)$24,564,195 $24,464,320 $1,626,833 V3_11.2_13.1  $(7.85) $(59,808,475)$24,564,195 $24,446,195 $1,625,628 V4_11.2_13.1  $(8.73) $(59,808,475)$27,768,221 $27,636,971 $1,837,809 V10_11.2_13.1 $(15.14) $(59,808,475)$39,516,314 $39,288,689 $2,612,627 V13_11.2_13.1 $(16.21) $(59,808,475)$39,516,314 $39,272,564 $2,611,555 V21_11.2_13.1 $(20.20) $(59,808,475)$41,652,331 $41,348,581 $2,749,606 V20_11.2_13.1 $(23.74) $(59,808,475)$42,720,339 $42,363,339 $2,817,086 V19_11.2_13.1 $(26.46) $(59,808,475)$43,788,348 $43,390,473 $2,885,389 V18_11.2_13.1 $(39.87) $(59,808,475)$50,196,399 $49,596,899 $3,298,105

Table 3 shows the classification performance statistics of parts 1 and 2of the cheating algorithm when used together to classify the 15,038subjects into impaired cheaters versus everyone else (normal cheatersand non-cheaters who passed the test when given by telephone). Thesensitivity is the accuracy of detecting impaired cheaters and thespecificity is the accuracy of detecting everyone else. To determinesensitivity, the sub-sample of 847 cheaters was used.

TABLE 3 Classification Performance Statistics for Different AlgorithmVersions Algorithm Version Sn Sp Acc PPV NPV V1_11.2_13.1 34% 91% 87%6.8% 98.6% V2_11.2_13.1 41% 88% 85% 6.6% 98.7% V3_11.2_13.1 41% 86% 84%5.7% 98.6% V4_11.2_13.1 46% 88% 85% 7.2% 98.8% V10_11.2_13.1 66% 83% 82%7.3% 99.2% V13_11.2_13.1 66% 81% 80% 6.6% 99.2% V21_11.2_13.1 70% 79%79% 6.3% 99.2% V20_11.2_13.1 71% 78% 78% 6.2% 99.3% V19_11.2_13.1 73%77% 77% 6.0% 99.3% V18_11.2_13.1 84% 64% 65% 4.4% 99.5%

The results show that there is a trade-off between sensitivity andspecificity, such that one cannot have both high sensitivity andspecificity. Considering the two tables together, there are greatersavings in future claims costs when one selects algorithms with a higherNegative Predictive Value, which means that fewer impaired cheaters weremisclassified. The tradeoff for this greater future claims savings is ahigher cost of paying for repeat testing in-person, which can go from alow of $5.25 per normal telephone test for the algorithm with thesmallest future claims savings, to a high of $39.87 per normal telephonetest for the algorithm with the greatest future claims savings.

Parts 1 and 2 of the Cheating Algorithm described above can be improvedby an analysis of the order of words recalled in each trial for impairedcheaters versus normal cheaters. Thus, analyzing a person's responses ona cognitive test, with respect to responses of a group of people, usinga classification algorithm can also involve evaluating a probability ofan order of items recalled by the person given probabilities of recallpatterns for the group of people. The recall patterns (across trials)for an individual can be compared with known recall patterns for a groupof people whose brain conditions are already established to a desiredlevel of accuracy. Given this information for the group of people, agood estimate of the conditional probability of recalling a particularset of items in a particular order can be determined for impaired versusnon-impaired individuals and for impaired cheaters versus non-impairedcheaters. This information can be compared with an individual's recallpattern(s) to determine whether or not the current individual likelycheated on the cognitive test.

The information for the group of people can be a set of well-classifiedcases generated in the following manner. A relatively large populationof subjects can be evaluated with an extensive neuropsychological testbattery, with functional measures, with severity staging measures (theClinical Dementia Rating Scale, the Functional Assessment Staging Test,and/or other measures), with laboratory testing and brain imaging. Theevaluated population is “relatively large” in the sense that there areenough cases to provide statistically significant results in light ofthe number of modeled categories, e.g., over four hundred subjects whenthe number of tuple categories (discussed further below) is sixteen. Theevaluated population should include normal subjects and subjects who areknown to have cheated on a given cognitive test.

Correspondence analysis can be used to analyze the cognitive testresults for the subjects (e.g., the binary score vectors of the trainingsample), and to compute the optimal row score matrix, optimal columnscore matrix and the singular value matrix. Correspondence analysis isan analytical method that has been largely used in quantitativeanthropology and the social sciences. Its primary function is tomaximize the canonical correlation between the rows and columns of aninput data matrix so that the maximum amount of information in the datacan be explained. Mathematically, it is designed to provide the bestlinear solution to the explanation of the information (variance) in thedata.

In some embodiments, correspondence analysis can be used to maximize theexplanation of the information that distinguishes impaired cheatersversus non-impaired cheaters. In the case of the CWL, the informationconsists of the patterns of recalled plus non-recalled words in eachtrial. In this sense, subject scores generated by correspondenceanalysis represent a complex combination of the subject characteristics(both normative and non-normative) plus word list test performancemetrics (e.g., words recalled, order recalled, retention time, etc.).The maximization of the explainable information can be accomplishedthrough a singular value decomposition of the input data matrix.

Correspondence analysis reduces the dimensionality of a raw data matrixwhile minimizing the loss of information. Tschebychev orthogonalpolynomials can be used to convert the raw data matrix into an optimalrow score matrix, an optimal column score matrix, and a singular valuematrix of eigenvalues. These matrices can have the following statisticalproperties: (1) each row of the optimal row score matrix consists of avector whose components are multivariate, normally distributed andstatistically independent of each other; (2) the optimal row scorevectors are also directly comparable because the effects of theirmarginal totals have been removed; (3) each column of the optimal columnscore matrix consists of a vector whose components are multivariate,normally distributed and statistically independent; (4) the optimalcolumn score vectors are also directly comparable because the effects oftheir marginal totals have been removed; (5) the singular value matrixconsists of a vector along the diagonal of the matrix, in which eachvalue represents a canonical correlation between the row and columnvariables of the optimal score matrices. Each value of the vector isstatistically independent of the other values, and indicates themagnitude of the contribution of each component of the optimal row andcolumn score vectors; the rank of these three matrices defines thenumber of statistically independent components needed to account for allof the explainable variance (non-noise) in the raw data. The rank isusually of much lower dimension than the number of rows or columns. Thismeans that the transformation of the input data matrix into a set ofstatistically orthogonal matrices (via singular value decomposition) canyield a massive reduction in dimensionality while continuing to accountfor most of the explainable information in the input data.

Thus, the optimal row scores represent the pattern of both recalled andnot recalled words in each trial after removing the effect of the totalnumber of words recalled, and the optimal column scores represent theeffects of recalling or not recalling a given word in a given trialafter removing the effect of the sample distribution. In this regard,the optimal row and optimal column scores are not simple weightings ofthe number of words recalled, their difficulty, their order or theirposition in the wordlist, or the specific sample used. Rather, theoptimal row and column scores provide the best linear solution toexplaining the total variance (information) of the raw data.

Correspondence analysis can thus produce optimal row and column scorevectors that only require a relatively small number of components (thefirst two or three components in many cases) to characterize themajority of the explainable variance of the input data matrix. Theseoptimal row and column score vectors can be derived by the simultaneousand inseparable use of the information from both normative andnon-normative cases as well as recalled and non-recalled words per trialto maximize data reduction and explanation of the total variance. Theoptimal column score and singular value matrices can be used forclassification of future subjects, while the optimal row score matrixcan be used to develop a statistical classification algorithm, such asone using logistic regression or discriminant analysis.

Various different cognitive tests and cognitive function scoringtechniques can be used. In any event, the cognitive test data can beanalyzed to identify a person as a likely cheater (230). Theidentification can be a Boolean indication or a number, such as ameasure of probability. The identification can be output to a device invarious manners, including displaying or printing a cheating indicationto an output device, transmitting the cheating indication over a networkto a computer system, or saving the cheating indication in acomputer-readable medium for use as input to further assessmentprograms.

As noted above, analyzing a person's responses on a cognitive test, withrespect to responses of a group of people, can involve evaluating aprobability of an order of items recalled by the person givenprobabilities of recall patterns for the group of people. The itemslearned and tested need not be words. However, the present disclosurefocuses on the case of the items being words, in the context of the CWL.This is done for purposes of clarity in this disclosure and in no waylimits the application of the systems and techniques described to thesespecific examples. In general, the described systems and techniques canbe used in any cognitive test in which the pattern of recall of an itemacross testing trials can be measured. Moreover, the described systemsand techniques can allow for variations in: (1) the number of learningtrials; (2) the number of testing trials; (3) the types of learningtrials used (e.g., presenting items visually or audibly, verifying ornot verifying that the subject correctly registered or understood theitem presented, providing cues for items not recalled, learning trialsin which the subject is presented only items not recalled in theprevious learning trial); (4) the types of testing trials (e.g., delayedcued recall versus delayed recognition versus delayed free recall,delayed free recall plus providing cues for items not recalled); (5) thenumber of items in the test list; (6) the number of items presented fromthe test list in each learning trial; and (7) the types of itemspresented in the test list (e.g., items presented as words, pictures orother visual displays, sounds, smells, tastes, and items presented bytouching them).

A recall pattern can be determined for each of multiple items across therecall trials. For example, subject test performance can be captured inthe following form; let:

$d_{ijk} = \left\{ \begin{matrix}1 & {{if}\mspace{14mu}{individual}\mspace{14mu} i\mspace{14mu}{responds}\mspace{14mu}{correctly}\mspace{14mu}{to}\mspace{14mu}{item}\mspace{14mu} j\mspace{14mu}{on}\mspace{14mu}{trial}\mspace{14mu} k} \\0 & {{if}\mspace{14mu}{individual}\mspace{14mu} i\mspace{14mu}{responds}\mspace{14mu}{incorrectly}\mspace{14mu}{to}\mspace{14mu}{item}\mspace{14mu} j\mspace{14mu}{on}\mspace{14mu}{trial}\mspace{14mu} k}\end{matrix} \right.$Then, the basic scoring element for the subject can be the responsevector:z _(ij)=(d _(ij1) ,d _(ij2) , . . . ,d _(ijK))where K is the total number of trials. There are 2^(K) possible responsetuples for each word. For the CERAD Wordlist, there are 16 (2⁴) responsetuples for each list word. Each of the 2^(K) possible response tuplesper word is assigned a unique response tuple value, c, which, for agiven subject's recall of that word across K trials is:

$c = {\sum\limits_{k = 1}^{K}\left( {2^{k}d_{ijk}} \right)}$Given a response tuple, c, the data can be coded as follows:

$x_{ijc} \equiv \left\{ \begin{matrix}1 & {{if}\mspace{14mu} c\mspace{14mu}{is}\mspace{14mu}{the}\mspace{14mu}{item}\mspace{14mu} j\mspace{14mu}{response}\mspace{14mu}{tuple}\mspace{14mu}{for}\mspace{14mu}{subject}\mspace{14mu} i} \\0 & {Otherwise}\end{matrix} \right.$As will be appreciated, this approach allows the response tuple value,c, to be used as a binary address within a computer system to accessx_(ijc), thus enabling more efficient processing. In any event, the goalcan then be to identify response tuples that optimize discriminationbetween impaired cheaters and non-impaired cheaters.

Many different types of classification algorithms can be applied to suchdata, including correspondence analysis, ordinal logistic regression,Bayesian hierarchical methods, and classification and regression trees.The example detailed below is based on discriminant analysis.

A probability of the recall patterns for the person can be evaluated,given probabilities of the recall patterns for the group of people. Forexample, suppose that for a given population, each word has a fixed setof probabilities of falling into the 2^(K) response tuples. Namely, fora given word, j, the prior probability response tuple vector, p_(jc), ofall possible response tuples is:P(x _(ijc)=1)=p _(j)→Multinomial(1;p _(j1) ,p _(j2) , . . . ,p _(jC)).Note that p_(j) is the prior probability response tuple vector thatwould be assigned to any subject for the given word, j, until moreinformation is known (such as the subject's performance for word j).Next, let the set consisting of the prior probability response tuplevectors for all list words be defined as the prior probability responsetuple profile, p, which equals <p_(1c), p_(2c), . . . , p_(Jc)>_(c=1)^(C). The implicit presumption here is that each word's probability ofrecall is independent of the other list words, which is why the words ina learning list should have low associability.

When a subject has performed the specified number of trials, K, one canthen compute their posterior probability response tuple profile, whichis:

${P\left( {D_{i}❘p} \right)} = {\prod\limits_{j = 1}^{M}{\prod\limits_{c = 1}^{C}p_{j}^{x_{ijc}}}}$D_(i)=<x_(ijc)>_(j=1) ^(M), represents the ith subject's response tuplefor each of the M list words, and p_(j) is the jth probability responsetuple vector for list word j across the K selected trials. Note that theterm, x_(ijc), equals 1 only for the response tuples, p_(jc), thatcharacterize the recall performance of the given subject, i, across thelist words.

The group membership of subject i (e.g., impaired cheater versusnon-impaired cheater) can be defined by an indicator variable, α_(i)where:

$a_{i} = \left\{ \begin{matrix}1 & {{if}\mspace{14mu}{subject}\mspace{14mu} i\mspace{14mu}{is}\mspace{14mu}{impaired}\mspace{14mu}{cheater}} \\0 & {{if}\mspace{14mu}{subject}\mspace{14mu} i\mspace{14mu}{is}\mspace{14mu}{non}\text{-}{impaired}\mspace{14mu}{cheater}}\end{matrix} \right.$Bayes theorem can be used to classify the subject to a particular groupby evaluating the probability of their response tuple profiles given theprobabilities of their response tuples:

$\begin{matrix}{{P\left( {a_{i} = {1❘D_{i}}} \right)} = \frac{{P\left( {a = 1} \right)}{P\left( {{D_{i}❘a} = 1} \right)}}{{{P\left( {a = 1} \right)}{P\left( {{D_{i}❘a} = 1} \right)}} + {{P\left( {a = 0} \right)}{P\left( {{D_{i}❘a} = 0} \right)}}}} & (1)\end{matrix}$Where P(α_(i)=1) and P(α_(i)=0) can be interpreted as the priorprobability of membership to impaired cheater and non-impaired cheatergroups respectively. In equation (1), the reliability of classifying agiven subject into the proper group depends upon the accuracy of theestimates of the response tuples, c, that are most relevant to groupdiscrimination. If there is a sufficiently large data set where thegroup membership, α, is known, then the estimated probability ofbelonging to a given group, α (e.g., impaired cheater versusnon-impaired cheater), for a given response tuple, c, and a given word,j, can be given by:

$\begin{matrix}{{\hat{p}}_{jca} = \frac{\sum\limits_{i = 1}^{N}{a_{i}\left( {x_{ijc} = c} \right)}}{N}} & (2)\end{matrix}$for a=0, 1 groups; i=1, . . . , N subjects; c=1, 2, . . . , 2^(K)response tuples; j=1, 2, . . . M words. Note that the termα_(i)(x_(ijc)=c) is set equal to “1” for all subjects belonging to thegroup being estimated. The group being estimated is made up of thoseindividuals whose recall pattern of the word, j, corresponds to theunique response tuple specified by the value, c, across the specifiedset of K trials.

Since the number of response tuples for any given word increasesexponentially with the number of trials, large samples may be needed toobtain reliable estimates of the response tuple profiles, p,particularly if some of the response tuples, c, are uncommon. For theCWL, there are four interesting combinations of trials that provide auseful dissection of memory performance. The first three immediate freerecall trials provide response tuples that measure working memoryperformance in the prefrontal cortex. The delayed free recall trialresponse tuples provide a measure of hippocampal storage and retrievalcombined. The delayed recognition trial response tuples provide ameasure of hippocampal storage. The first four trials or all five trialscombined provide overall measures of memory performance.

For the four-trial CWL response tuples, one needs thousands of cases toobtain adequate estimates of each possible response tuple for each word.A database of cases can be built for this purpose, in which groupmembership is not explicitly known but can be reasonably accuratelyestimated by a previously established, validated algorithm (see e.g.,Cho, et al., “Early Detection and Diagnosis of MCI Using the MCI ScreenTest,” The Japanese Journal of Clinical and Experimental Medicine, 2007;84(8):1152-1160; Trenkle, et al., “Detecting Cognitive Impairment inPrimary Care: Performance Assessment of Three Screening Instruments,”Journal of Alzheimer's Disease, 2007; 11(3):323-335; and Shankle et al.,“Method to improve the detection of mild cognitive impairment”, PNAS,Vol. 102, No. 13, pp. 4919-4924, 2005).

Group membership of each case in the database can be independentlydetermined twice by the algorithm, first using a high sensitivitycut-point (e.g., Sn=96%, Sp=88%), which can identify a relatively puresample of non-impaired cheaters, and then using a high specificitycut-point (e.g., Sn=83%, Sp=98%), which can identify a relatively puresample of impaired cheaters. The performance of the group membershipprobability estimates derived from equation (2) can then be evaluated byeach of these two cut-points for each response tuple of each word. Thisevaluation can be accomplished using each set of probability estimatesindependently to classify a different sample of subjects with knowngroup membership. Note that an implicit presumption of this method isthat the classification error attributable to the previously establishedalgorithm is random relative to the response tuples.

FIG. 4 shows another exemplary system 400 used to identify cheaters on acognitive test. The exemplary system described can perform a variety offunctions including data analysis, storage and viewing, and remoteaccess and storage capabilities useful for analyzing results of acognitive test, including identifying cheaters on the cognitive test,such as described elsewhere in this specification.

A Software as a Service (SaaS) model can provide network based access tothe software used to identify cheaters on a cognitive test. This centralmanagement of the software can provide advantages, which are well knownin the art, such as offloading maintenance and disaster recovery to theprovider. A user, for example, a test administrator within a clinicalenvironment 410, can access test administration software within the testadministration system via a web browser 420. A user interface module 430receives and responds to the test administrator interaction.

In addition, a customer's computer system 440 can access software andinteract with the test administration system using an eXtensible MarkupLanguage (XML) transactional model 442. The XML framework provides amethod for two parties to send and receive information using astandards-based, but extensible, data communication model. A web serviceinterface 450 receives and responds to the customer computer system 440in XML format. For example, an XML transactional model can be useful forstorage and retrieval of the structured data relating to the cognitivetest.

An analysis module 460 analyses inputs from the web service interface450 and the user face module 430, and produces test results to send. Theanalysis module uses a results analysis module 470 to perform the testanalysis to identifying cheaters. The results analysis module 470 can,for example, incorporate the methods described elsewhere in thisspecification.

A data storage module 480 transforms the test data collected by the userinterface module 430, web service interface 450, and the resulting datagenerated by the analysis module 460 for permanent storage. Atransactional database 490 stores data transformed and generated by thedata storage module 480. For example, the transactional database cankeep track of individual writes to a database, leaving a record oftransactions and providing the ability to roll back the database to aprevious version in the event of an error condition. An analyticaldatabase 492 can store data transformed and generated by the datastorage module 480 for data mining and analytical purposes.

Embodiments of the subject matter and the functional operationsdescribed in this specification can be implemented in digital electroniccircuitry, or in computer software, firmware, or hardware, including thestructures disclosed in this specification and their structuralequivalents, or in combinations of one or more of them. Embodiments ofthe subject matter described in this specification can be implemented asone or more computer program products, i.e., one or more modules ofcomputer program instructions encoded on a tangible program carrier forexecution by, or to control the operation of, data processing apparatus.The tangible program carrier can be a propagated signal or acomputer-readable medium. The propagated signal is an artificiallygenerated signal, e.g., a machine-generated electrical, optical, orelectromagnetic signal that is generated to encode information fortransmission to suitable receiver apparatus for execution by a computer.The computer-readable medium can be a machine-readable storage device, amachine-readable storage substrate, a memory device, or a combination ofone or more of them.

The term “data processing apparatus” encompasses all apparatus, devices,and machines for processing data, including by way of example aprogrammable processor, a computer, or multiple processors or computers.The apparatus can include, in addition to hardware, code that creates anexecution environment for the computer program in question, e.g., codethat constitutes processor firmware, a protocol stack, a databasemanagement system, an operating system, a cross-platform runtimeenvironment, or a combination of one or more of them. In addition, theapparatus can employ various different computing model infrastructures,such as web services, distributed computing and grid computinginfrastructures.

A computer program (also known as a program, software, softwareapplication, script, or code) can be written in any form of programminglanguage, including compiled or interpreted languages, declarative orprocedural languages, and it can be deployed in any form, including as astand-alone program or as a module, component, subroutine, or other unitsuitable for use in a computing environment. A computer program does notnecessarily correspond to a file in a file system. A program can bestored in a portion of a file that holds other programs or data (e.g.,one or more scripts stored in a markup language document), in a singlefile dedicated to the program in question, or in multiple coordinatedfiles (e.g., files that store one or more modules, sub-programs, orportions of code). A computer program can be deployed to be executed onone computer or on multiple computers that are located at one site ordistributed across multiple sites and interconnected by a communicationnetwork.

The processes and logic flows described in this specification can beperformed by one or more programmable processors executing one or morecomputer programs to perform functions by operating on input data andgenerating output. The processes and logic flows can also be performedby, and apparatus can also be implemented as, special purpose logiccircuitry, e.g., an FPGA (field programmable gate array) or an ASIC(application-specific integrated circuit).

Processors suitable for the execution of a computer program include, byway of example, both general and special purpose microprocessors, andany one or more processors of any kind of digital computer. Generally, aprocessor will receive instructions and data from a read-only memory ora random access memory or both. The essential elements of a computer area processor for performing instructions and one or more memory devicesfor storing instructions and data. Generally, a computer will alsoinclude, or be operatively coupled to receive data from or transfer datato, or both, one or more mass storage devices for storing data, e.g.,magnetic, magneto-optical disks, or optical disks. However, a computerneed not have such devices. Moreover, a computer can be embedded inanother device, e.g., a mobile telephone, a personal digital assistant(PDA), a mobile audio or video player, a game console, a GlobalPositioning System (GPS) receiver, or a portable storage device (e.g., auniversal serial bus (USB) flash drive), to name just a few. Devicessuitable for storing computer program instructions and data include allforms of non-volatile memory, media and memory devices, including by wayof example, semiconductor memory devices, e.g., EPROM, EEPROM, and flashmemory devices; magnetic disks, e.g., internal hard disks or removabledisks; magneto-optical disks; and CD-ROM and DVD-ROM disks. Theprocessor and the memory can be supplemented by, or incorporated in,special purpose logic circuitry.

While this specification contains many implementation details, theseshould not be construed as limitations on the scope of the invention orof what may be claimed, but rather as descriptions of features specificto particular embodiments of the invention. Certain features that aredescribed in this specification in the context of separate embodimentscan also be implemented in combination in a single embodiment.Conversely, various features that are described in the context of asingle embodiment can also be implemented in multiple embodimentsseparately or in any suitable subcombination. Moreover, althoughfeatures may be described above as acting in certain combinations andeven initially claimed as such, one or more features from a claimedcombination can in some cases be excised from the combination, and theclaimed combination may be directed to a subcombination or variation ofa subcombination.

Similarly, while operations are depicted in the drawings in a particularorder, this should not be understood as requiring that such operationsbe performed in the particular order shown or in sequential order, orthat all illustrated operations be performed, to achieve desirableresults. In certain circumstances, multitasking and parallel processingmay be advantageous. Moreover, the separation of various systemcomponents in the embodiments described above should not be understoodas requiring such separation in all embodiments, and it should beunderstood that the described program components and systems cangenerally be integrated together in a single software product orpackaged into multiple software products.

Thus, particular embodiments of the invention have been described. Otherembodiments are within the scope of the following claims. For example,the actions recited in the claims can be performed in a different orderand still achieve desirable results. The actions recited in the claimscan be performed using different statistical classification procedures,such as discriminant analysis, stepwise multivariate regression orgeneral linear models, rather than logistic regression as describedabove. The actions recited in the claims can be performed usingdifferent orthogonal transformations of the raw input data, such asprincipal components analysis, multi-dimensional scaling, or latentvariable analysis, rather than correspondence analysis as describedabove.

Moreover, additional techniques can be employed in combination withthose described above. Cheaters from non-cheaters can be distinguishedbased on additional checks for similarities in the pattern ofperformance. Statistical techniques can be used to detect strangepatterns in the data. Some examples include techniques to detectoutliers, order effects, abnormal dispersion, and spurious correlations.Use of such techniques in combination with the system and techniquesdescribed above employs the premise that it is difficult to inventnaturalistic data, particularly those with multiple dimensions.

What is claimed is:
 1. A computer-implemented method comprising:receiving first information concerning a person, the first informationspecifying the person's responses, and lack thereof, for items of acognitive test administered to the person, wherein the cognitive testcomprises multiple item-recall trials used to assess cognitiveimpairment; analyzing the first information using a classificationalgorithm trained on second information concerning a group of people towhom the cognitive test has been administered, the classificationalgorithm generated in accordance with a first part and a second part,the first part distinguishing between cheaters and non-cheaters, and thesecond part distinguishing between impaired cheaters and non-impairedcheaters; and identifying, based on the analyzing, the person as acheater requiring a verification test to confirm an initial result ofthe cognitive test.
 2. The method of claim 1, wherein the classificationalgorithm is configured to check for cheating strategies characteristicof persons with Alzheimer's disease or a related disorder (ADRD).
 3. Themethod of claim 1, wherein the classification algorithm is selected tomaximize sensitivity while minimizing reduction in specificity, whichpreserves a high negative predictive value while maintaining a lowmisclassification rate for impaired cheaters, and the group of peoplecomprising a first sample and a second sample, the method furthercomprising: analyzing data in the first part for the first sample toidentify a subset of variables that discriminate between cheaters andnon-cheaters, and validating results in the first part using the secondsample; and analyzing, in the second part, data of persons identified ascheaters in the first part to identify a subset of variables thatdiscriminate between impaired cheaters and non-impaired cheaters.
 4. Themethod of claim 3, wherein the subset of variables that discriminatebetween cheaters and non-cheaters comprises education and multiplenumbers corresponding to items recalled on two or more of the multipleitem-recall trials including a delayed free recall trial, and the subsetof variables that discriminate between impaired cheaters andnon-impaired cheaters comprises age, gender, education and a numbercorresponding to items recalled on at least one of the multipleitem-recall trials.
 5. The method of claim 1, wherein the multipleitem-recall trials comprise word recall tests of memory, and theanalyzing comprises distinguishing between impaired and non-impairedindividuals based on a total number of words recalled across the trials.6. The method of claim 1, wherein the analyzing comprises evaluating aprobability of an order of items recalled by the person givenprobabilities of recall patterns for the group of people.
 7. Anon-transitory computer-readable medium encoding a computer programproduct operable to cause data processing apparatus to performoperations comprising: receiving first information concerning a person,the first information specifying the person's responses, and lackthereof, for items of a cognitive test administered to the person,wherein the cognitive test comprises multiple item-recall trials used toassess cognitive impairment; analyzing the first information using aclassification algorithm trained on second information concerning agroup of people to whom the cognitive test has been administered, theclassification algorithm generated in accordance with a first part and asecond part, the first part distinguishing between cheaters andnon-cheaters, and the second part distinguishing between impairedcheaters and non-impaired cheaters; and identifying, based on theanalyzing, the person as a cheater requiring a verification test toconfirm an initial result of the cognitive test.
 8. The non-transitorycomputer-readable medium of claim 7, wherein the classificationalgorithm is configured to check for cheating strategies characteristicof persons with Alzheimer's disease or a related disorder (ADRD).
 9. Thenon-transitory computer-readable medium of claim 7, wherein theclassification algorithm is selected to maximize sensitivity whileminimizing reduction in specificity, which preserves a high negativepredictive value while maintaining a low misclassification rate forimpaired cheaters, and the group of people comprising a first sample anda second sample, the operations further comprising: analyzing data inthe first part for the first sample to identify a subset of variablesthat discriminate between cheaters and non-cheaters, and validatingresults in the first part using the second sample; and analyzing, in thesecond part, data of persons identified as cheaters in the first part toidentify a subset of variables that discriminate between impairedcheaters and non-impaired cheaters.
 10. The non-transitorycomputer-readable medium of claim 9, wherein the subset of variablesthat discriminate between cheaters and non-cheaters comprises educationand multiple numbers corresponding to items recalled on two or more ofthe multiple item-recall trials including a delayed free recall trial,and the subset of variables that discriminate between impaired cheatersand non-impaired cheaters comprises age, gender, education and a numbercorresponding to items recalled on at least one of the multipleitem-recall trials.
 11. The non-transitory computer-readable medium ofclaim 7, wherein the multiple item-recall trials comprise word recalltests of memory, and the analyzing comprises distinguishing betweenimpaired and non-impaired individuals based on a total number of wordsrecalled across the trials.
 12. The non-transitory computer-readablemedium of claim 7, wherein the analyzing comprises evaluating aprobability of an order of items recalled by the person givenprobabilities of recall patterns for the group of people.
 13. A systemcomprising: a user device; and one or more computers operable tointeract with the device and to perform operations comprising: receivingfirst information concerning a person, the first information specifyingthe person's responses, and lack thereof, for items of a cognitive testadministered to the person, wherein the cognitive test comprisesmultiple item-recall trials used to assess cognitive impairment;analyzing the first information using a classification algorithm trainedon second information concerning a group of people to whom the cognitivetest has been administered, the classification algorithm generated inaccordance with a first part and a second part, the first partdistinguishing between cheaters and non-cheaters, and the second partdistinguishing between impaired cheaters and non-impaired cheaters; andidentifying, based on the analyzing, the person as a cheater requiring averification test to confirm an initial result of the cognitive test.14. The system of claim 13, wherein the classification algorithm isconfigured to check for cheating strategies characteristic of personswith Alzheimer's disease or a related disorder (ADRD).
 15. The system ofclaim 13, wherein the classification algorithm is selected to maximizesensitivity while minimizing reduction in specificity, which preserves ahigh negative predictive value while maintaining a low misclassificationrate for impaired cheaters, and the group of people comprising a firstsample and a second sample, the operations further comprising: analyzingdata in the first part for the first sample to identify a subset ofvariables that discriminate between cheaters and non-cheaters, andvalidating results in the first part using the second sample; andanalyzing, in the second part, data of persons identified as cheaters inthe first part to identify a subset of variables that discriminatebetween impaired cheaters and non-impaired cheaters.
 16. The system ofclaim 15, wherein the subset of variables that discriminate betweencheaters and non-cheaters comprises education and multiple numberscorresponding to items recalled on two or more of the multipleitem-recall trials including a delayed free recall trial, and the subsetof variables that discriminate between impaired cheaters andnon-impaired cheaters comprises age, gender, education and a numbercorresponding to items recalled on at least one of the multipleitem-recall trials.
 17. The system of claim 13, wherein the multipleitem-recall trials comprise word recall tests of memory, and theanalyzing comprises distinguishing between impaired and non-impairedindividuals based on a total number of words recalled across the trials.18. The system of claim 13, wherein the analyzing comprises evaluating aprobability of an order of items recalled by the person givenprobabilities of recall patterns for the group of people.
 19. The systemof claim 13, wherein the one or more computers comprise a server systemoperable to interact with the device through a data communicationnetwork, and the device is operable to interact with the server as aclient.
 20. The system of claim 13, wherein the device comprises a userinterface device, the one or more computers comprise the user interfacedevice, and the operations further comprise outputting an indication ofthe identified person to a device comprising a computer-readable medium.